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Overview (U) 



• (U) Setting the Stage 

Strategic Surprise, Priority Needs, Definitions 

• (U) Making Things Measurable 

- Emerging Technology Discovery 

- Technology Use Discovery 

• (U) Challenges 

- Complexity 

- Getting data is only step 1 

- Visualization 

- Building outreach and engagement 
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CT Trends Focus Questions (U) 



(U) Does NSA CT know what technologies, 
communications products and applications, and 
modus operandi are being used by terrorists, 
terrorist groups, or in locations of interest? 



(U) Does NSA CT know what emerging 
technologies, communications products and 
applications, and modus operandi are likely to be 
used by terrorists, terrorist groups, or in locations 



of inl-orocl-? 

Prevent Strategic 
Surprise 
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CT Trends Focus Questions (U) 

(U) Does NSA CT know what technologies, 
communications products and applications, and 
modus operandi are being used by terrorists, 
terrorist groups, or in locations of interest? 

(U) Does NSA CT know what emerging 
technologies, communications products and 
applications, and modus operandi are likely to be 
used by terrorists, terrorist groups, or in locations 

(C//RtL) What^ipWWeifly asking is: 

Can we tell which ones are likely to become 
a priority needZ 
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Risk Management for SIGINT 

Threats (U) 

• (S//REL) Threat to SIGINT Capability 

- A behavior or technology that has the potential to have a negative 
impact on NSA's capability to provide SIGINT on a Terrorism Target 

• (U) Use Risk 

- The possibility that a particular threat will be adopted by Terrorist targets 

• (S//REL) Indications and Warning 

- Early warning of high impact threats to prevent surprise to key 
stakeholders and reduce risk from Terrorist adoption of technology that 
would adversely affect SIGINT production 



' (5//REL) NSA's ability to manage risk N 
is directly proportional to our ability 

< to detect threats > 
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The data-driven approach 

(U) 

"Count what is countable, measure what is measurable, and make measurable that 

which cannot be measured" 

Galileo (17th century astronomer) 



"When you can measure what you are speaking about, and express it in numbers, 
you know something about it; but when you cannot measure it, when you cannot 
express it in numbers, your knowledge is of a meagre and unsatisfactory kind" 

Lord Kelvin (discovered absolute zero) 



" You cannot manage what you cannot measure " 
Bill Hewlett (co- founder of Hewlett-Packard) 



"Not everything that counts can be counted, and not everything that can be counted 

counts" 

- Albert Einstein 
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So... what is a (CT) trend? 



A trend is a measurement of occurrence 

(S//REL) Comparing the behavior of a single target... 

- Pattern -of- life 

- Modus Operandi 

- Technology Usage 



...to the behaviors seen within the target space 



Multiple targets, within and across the entire CT enterprise 

- Over a period of time 






TOP SECRET//COMINT//REL FVEY//20340601 



Prediction and Identification of 
Priority Needs Prevents Strategic 

Surprise (U) 




Known 

< 



Identify issues that are 
emerging into and 
using, wjthin the target 
space 



Rising 



Emergmt 



TOP SECRET//COMINT//REL FVEY//20340601 





TOP SECRET//COMINT//REL FVEY//20340601 



Making Things Measurable 




Te§hn§i@|i§§ Theughi Leaders § in U§§ 
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Innovation Phases (U) 



Adoption 



Experimentation 



Interest 
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Technology Adoption Factors 

(U) 
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Optics (U) 



• (S//REL) Optic #1: Emerging Technology Discovery 

Focused primarily on interest and experimentation phases of innovation 

- Watching the Watchers 

- Weaker indicators 

New technologies 



• (S//REL) Optic #2: Technology Use Discovery 

Focused primarily on adoption phase of innovation 

- Owning the Known 

- Stronger indicators 

New targets 



12 
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Analytics and Processes (U) 




Production Element 

Crarnimn Mnla Pfniarf 



FORTREND - Extremist 

Tarhniral Ci ih-Cnri im 



Administrative Response 

Tor hnnlnmr I lea Hicr awoki# 



T 



Production Element 

Cramtirin PKAiQ^I- 



FORTREND - Extremist 

Tnrhnii-al 



Production Element 

- Sran ninn Mn tfl Pcoiact 

FORTREND - Extremist 

Tarh»ii-al Cnh-CnKum 



Administrative Response 

TarhnnlAnw I lea nicrAuoni 



Administrative Response 

Tnr knnlnriir Ilea niernuon 






Seized Media 

Tranrlc TrarUnn 



CNE Trend Tracking 

a wrl nicrA\/ar , \f 





Technology Pattern of 



Mobile 




Technology Pattern of 



Mobile 




Technology Pattern of 

Mobile 
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Optic #1: Emerging Technology 

Discovery (U) 

• (S//SI//REL) Emerging Technology & Behavior Discovery 

Detection of interest, experimentation, knowledge transfer or direction using 
content, metrics approaches 

Currently using deskside & virtual engagement to leverage TOPI analyst 
initiative to discover, prioritize, and work against “strongest” indicators 

Leverages inherent TOPI expertise and functions of traffic processing/translation/tasking 
etc.. 

- Embedded analysts, virtual relationships: production "customers" 

- Currently identifying, tracking 'technical' thought leaders 

- Technical sub-forums, scanning notes measurements 

- Administrative emails (No-Reply etc..) 

- Forum links, uploaded/downloaded files 



Coal: Generate Prioritized Input (techs/behaviors) for Research 
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Optic #2: Technology Use 

Discovery (U) 

. (S//SI//REL) Technology Use and Behavior Discovery 

_ "Stratactical" data sets 

_ Includes target-specific data point for each item (e.g. selector) 

_ Discovery of target behavior by identifying technology use patterns, trends, and/or 
anomalies in: 

_ User-agents (browsers, OS, devices) 

_ Tasking (new tasking, total tasking) 

_ Network, Protocol usage (Active User metrics) 

_ Visited URLs, web searches 

_ Process lists, pre-fetch logs, registry entries, software logs 
_ Hardware usage (smartphones, tablets, SD cards) 

_ Currently using various tools (XKEYSCORE, SEEKER, BIONICTURTLE, JEMA, JOLLYROGER, 
MARINA, TUNINGFORK, QFDs, etc...) and approaches with multiple cloud analytics in 
varying stages of development and/or planning 

Goal: Generate Prioritized Input (techs/behaviors) for Research 

15 
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Measurement Drives Research (U) 

(S//REL) Triage begins with target indicators of a new technology 

Derived from either optic: Emerging or Use Discovery 

Interest, Experimentation, Use, Knowledge Transfer, Metric, etc... 

Target Technology n Do other targets use this technology? 

This is the central defining question for Trends Analysis : 

Do other CT targets use this technology? 
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Weak vs. Strong Indicators: Brutal 

Triaae (U) 




Exper Exper Exper 

Previous/Low 1 Previous/Low I Previous/Low 

Installed, no Installed, no Installed, no 




Log files, traffic Log files, traffic Log files, traffic 
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The Wicked Problem Aspect (U) 

(S//REL) Defining the problem is the first (wicked) 

problem 

- Triage Stage 1 

- Initial priority: (single) target + initial understanding of technology 

- Implications Research 

- What does the product/service do? 

- Current NSA capabilities to detect, collect, exploit, analyze? 

- Do any other CT targets use it? 

- Triage Stage 2 

- Updated priority: target(s) + updated understanding of tech/USSS 

- Validated, Next Steps 

- As needed: capabilities/access development requirements 

- Reporting: internal, CIR, e-gram; Gaps report; prioritization w/in tech category 

18 
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Goal: Periodic Reporting 

Vehicle (U) 

. (U//FOUO) Move beyond ad hoc task responses to routine deliverables 

. (U//FOUO) Overcoming volume challenge 
_ Fluge variety of inputs, massive numbers in each 
_ Prioritization 
_ Visualization 

• (S//REL) Moving threats to a simple Risk Assessment model 

_ Borrows methodology from models used for executive purposes elsewhere in agency 
- (FAMT, Geopolitical Technology Trends Matrix, TAO...) 

_ Opportunities, threats handled separately 
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Capabilities Development Risk 



Impact 

> 

to production 

Use 

Risk 

v 



Current 

Highest Priority 
Target Use 

Current 
Operational 
Target Use 

Current Low 
Priority/Previou 
s Higher 
Priority Target 
Use 

Technical 
Thought Leader 
Recommendati 
ons, 

Pvnorimonfafin 



MaEriv l\ H 



TRIVIAL 


MINOR 


MODERATE 


MAIOR 


CATASTROPHIC 


Loss/lack of 
insight to small 
aspect of target 
communications 
, presence 


Loss/lack of 
insight to 
significant 
aspect of 
target 

communication 
s, presence 


Loss/lack of 
insight to large 
component of 
target 

communications, 

presence 


Loss/lack of 
insight to 
majority of 
target 

communications 
, presence 


Near-total 
loss/lack of 
insight to target 
communications 
, presence 


Document 

tracking 


Fivewes, 

Facebook 

chat 

presentation 


Mail.ru, 
Team Viewer, 
Join. me 


OTR, Tor, 
Smartphones, 
Zoho.com 
webmail, 
TrueCrypt 


Tor+ Trilight 
Zone + Cspace 
+ ZRTP VoIP 
client on Linux 
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Capabilities Development 
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Capability Development 
Challenges (U) 

(TS//SI//REL) With rare exceptions, 
a pplication-specific solutions are only 
built based on these two criteria???? 

. In resource-restrained environment, development of capabilities against likely- 
to-increase in priority applications is trumped by standing requirements 
driven by known priority applications 



. Capabilities development response to current/priority technology threats occurs 
normally w/in existing resources - but response does not scale, either to the 
industry or to multiple crises 
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Simplifying the Risk Matrix (U) 



TRIVIAL 

Loss/lack of 
insight to small 
aspect of target 
communications 
, presence 



Impact 

> 

to production 

Use 

Risk 



Current 

Highest Priority 
Target Use 



Current 
Operational 
Target Use 

Current Low 
P r i o r ity/P re vi o u 
s Higher 
Priority Target 
Use 

Technical 
Thought Leader 
Recommendati 
ons, 

Experimentatio 

n 
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Adding in the Solution 
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Examples: Jan-February 2012 




MAIOR 



Muslima 

Purematrimony 

com 

Zemana Anti- 
Keylogger 



Web.de 



Impact 

> 

to production 

Use 

Risk 

v 
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Current 
Operational 
Target Use 
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Goal: Emerging Technology 

Snapshot (U) 

• (U) Executive version - snapshot of top items only 

• (S//REL) Overcoming the challenges of prioritization and volume 
is still only 50% of the problem 

• (S//REL) Stated Preference: 

- Breakdowns by target/target set 

- Preserve opportunity vs. threat 

- Identify HUMINT sources for collaboration 
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Emerging Technology 
Snapshot (U) 



Target/Org 


Tech 


Quadrant 


AQSL 

courier 


TAILS 




GIMF 


TAILS 


1 


AQ media 


TrueCrypt 




S2I42 


Join. Me 




LT, S2I42 


TeamViewer 


2 


LT 


Laplink 




TTL 


Extremist version 
of Tor 


Opportu 

nity 


AQ media 


Encrypted Webmail 


Source 



(TS//SI//REL) Full details available as needed 
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Emerging Technology 
Snapshot (U) 

• (S//REL) Monthly Emerging Technology Snapshot 

- 1-3 page Snapshot (6 page max if previous month data 
included) to CT leadership 

- Snapshot + supporting full data to MICROEXPANSE 
Underlying processes in alpha stage 

Stopgap until maturation of multiple efforts 

- Data Explorer, ECHOBASE 

- Inclusion of FAA/PRISM in GM-Halo 
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End Results - Tactical & 

Strategic (U) 

. (S//REL) Tactical Outcomes 

- Lead Generation 

- Target Development 

- Target Discovery 

_ Behavior Detection 

- Access Prioritization 



. (S//REL) Strategic Outcomes 

_ Prioritization for Capabilities Development 

_ Driven by target priority: single target + volume of targets 
_ Prioritized within tech category, target (set) category 
- Overall CT product line prioritization 
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Challenges (U) 



(C//REL) Complexity 

_ Understand target, technology, & SIGINT system 

(S//SI//REL) Getting data is only step 1 

_ Getting a data set is like to getting a new bearer to analyze 

(U) Visualization 

_ Excel tops out at a million rows... 

(TS//SI//REL) Clean data 

_ Targets vs. Selectors 
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Overcoming Complexity (U) 

SIGINT 1 b y stem 

Fingerspitzengefiihl 

• Literally "fingertip feeling" 

• Empathy, sensitivity, tact 

• Ability of military commanders 
to react rapidly 



CT Trends Tearnthnolog^ crinolo 9y 

SIGDEV analysts 
Partner/Enablers 



Must understand tech threat 
implications, provenance and 
structure of data to manipulate, 
interpret it 
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Getting Data is Step 1 (U) 
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Getting Data is Step 1 (U) 
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Getting Data is Step 1 (U) 
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Visualization (U) 



(TS//SI//REL) Excel tops out at a million rows... 

- 19 branches, 30+ target sets, -200 realms, -800 domains, -45000 
selectors = 1 million rows/~2.5 weeks for summarized active user events 
from E012333 alone 

- Spreadsheets are good, but not everyone knows how to use a 
pivot table 

- Each dataset can easily provide 4-5 or more pivoted looks for 
each branch/target set = minimum 100-150 slides 



s (S//REL) Intent is to routinely produce^ 
multiple large datasets on a monthly 
basis for collection management, 

^ research purposes 
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Visualization (U) 

(S//REL) Analysts work at the selector level 
• Leadership wants data presented at the target level 



(S//REL) Automated population of technology, behavior 
information in analyst workflow tools, databases 



(S//REL) Each separate visualization task takes 
manpower, time away from operational analysis 
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Clean Data (U) 



(S//SI//REL) Metrics will only provide a near-accurate picture: ground truth will always be the domain of 
the TOPI and based on content 



(S//SI//REL) Some selectors (accurately) map to multiple targets, multiple teams, multiple organizations 



(S//SI//REL) Some selectors simply don't have a known target, only a target set 



(S//REL) Need to correlate across widely different datasets requires creation of normalized bridge 
datasets (e.g. comparing executables to domains) 



(S//SI//REL) TKB/UTT are victims of years of "fill in the blank" freeform data entry; very slowly being 
addressed (-2015?) 
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Rising Strategic Issues (U) 



• (TS//SI//REL) Encrypted Webmail Services 

- Atabmail, Zoho, Safe-mail, Fastmail, HMA Mail 

• (TS//SI//REL) Remote Desktop Viewers/Remote Access Tools 

- TeamViewer, Join. me. Cybergate 

• (TS//SI//REL) Aggregators/Over-the-Top Messaging Services 

- WhatsApp, Nimbuzz, eBuddy 
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What Next? (U) 



(S//SI//REL) Continue to build, strengthen, expand: 

_ internal workflows, research and discovery capabilities 

_ collaboration with production elements 
_ Operational support via embedded analysts at NSAW 
_ Tradecraft, technical support virtually with extended enterprise 

_ partnerships with FVEY SIGDEV community 

_ Establish and expand dialogue opportunities 

"Failure Sharing" - tradecraft sharing and operational deconfliction 



. (S//REL) Technology Trends MyNoc 
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Question 

s? 
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